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This  report  describes  the  progress  made  under  contract  N00014-75-C-0266, 
project  no.  NR  049-345  during  the  period  January  1,  1976  - March  31,  1977. 
Included  are  the  description  of  the  reports  distributed,  activities  under- 
taken, and  activities  of  the  personnel  supported  by  the  contract. 
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The  reports  produced  during  this  period  included  the  following: 

1.  An  Operations  Research  Approach  for  the  'Design  to  Cost' 
of  a Complex  Radar  System 

2.  Evaluation  of  Network  Services 

3.  Experiments  in  Minicomputer  Networks 

4.  A Comparative  Evaluation  of  Versions  of  BASIC 

5.  Modified  Bayesian  Procedures  in  Reliability  Testing 

6.  Management  Evaluation  of  Network  Performance  with  Security 
Measures 

7.  Computing  Auditing:  The  Problems  vs.  SAS  no.  3 

8.  Characteristics  of  Application  Software  Maintenance 

9.  Maintenance  Questionnaire  and  List  of  Respondents 

10.  Description  of  Computer  Network  Analysis  Model 

' v 

The  first  report  is  a reprint  of  an  article  that  appeared  in  the  Israel 
Journal  of  Engineering  (April  1976;  15-24).  The  second  paper  was  given  as 
an  invited  paper  at  the  Annual  meeting  of  the  ACM  and  appeared  in  the  Conference 
Proceedings  (ACM  76,  218-220).  The  third  paper  is  a technical  report  on  some 
of  the  experiments  conducted  for  the  Naval  Electronics  Laboratory  Center  in 


1976.  This  work  is  continuing  and  will  be  described  below.  The  fourth  paper 

j 

is  a reprint  of  an  article  that  appeared  in  the  Communications  of  the  ACM 
(April,  1976,  175-181).  The  fifth  paper  is  a chapter  in  the  book  Theory 
and  Applications  in  Reliability  (C.  P.  Tsokos,  ed.,  1977  - in  press),  ^he 
sixth  paper  was  presented  as  an  invited  paper  at  the  Eurocomp  Conference  on 

i ! 

Computer  Performance  and  appears  as  a chapter  in  Computer  Performance 
Evaluation,  On  line  Publishing  (1976,  651-658).  The  seventh  paper  is  a 


technical  report  and  is  scheduled  to  appear  in  the  March,  1977  issue  of  the 
CPA  Journal.  The  eighth  and  ninth  papers  are  results  of  new  work  in  the 
area  of  software  maintenance.  The  eighth  paper  has  been  submitted  to  the 
Communications  of  the  ACM.  The  tenth  paper  is  a technical  report  describing 
the  computer  network  model  used  in  the  analysis. 

A major  area  of  interest  during'  the  period  was  the  analysis  of  application 
software  maintenance.  During  this  period  the  literature  was  reviewed,  a 
questionnaire  prepared,  distributed,  and  analyzed.  The  questionnaire  and 
list  of  respondents  appear  in  [9].  The  results  of  analyzing  the  maintenance 
questionnaire  data  are  presented  in  [8].  The  questionnaire  was  pretested 
and  then  distributed  to  120  organizations  located  on  the  West  Coast.  Responses 
totaled  69  at  the  time  of  writing  the  paper.  The  total  responses  to  date 
have  been  78.  The  results  of  the  analysis  are  presented  in  detail  in  [8]. 

The  results  will  be  briefly  summarized  here. 

Very  little  research  has  been  done  previously  in  this  area.  This  is  somewhat  sur- 
prising since  it  has  been  cited  that  over  60%  of  the  total  life  cycle  cost  of  the 
system  occur  in  maintenance  and  enhancements.  Furthermore,  sources  cite  that 
systems  groups  spend  almost  50%  of  their  efforts  on  maintenance  and  enhance- 
ment. 


The  effort  in  this  area  is  being  directed  toward  determining  problem  areas 
and  what  is  being  done  in  regard  to  maintenance.  After  this  has  been  done, 
analysis  and  modeling  methods  can  be  used  for  examining  maintenance.  To 
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determine  properties  and  characteristics  of  maintenance  a questionnaire  was 
developed.  The  questionnaire  is  in  two  parts.  Part  I deals  with  the  systems 
and  programming  department  and  contains  12  questions  in  the  following  areas: 

- industry  category 

- annual  budget  for  software  and  hardware 

- number  of  personnel  in  department  (systems  analysts  and 
programmers  as  well  as  aggregate) 

- division  of  tasks  among  staff  in  maintenance  and  new 
application  work,  and  in  analysis  and  programming 

- management  structure 

- current  percentage  of  effort  in  maintenance 

- relative  importance  of  maintenance  compared  to  development 

- reallocation  of  effort  between  maintenance  and  development, 
given  hypothetical  budget  increases  and  decreases 

- evaluation  of  adequacy  of  current  levels  of  staffing 

The  second  part  of  the  questionnaire  dealt  with  the  application  software 
undergoing  maintenance  and  enhancement.  Respondents  were  asked  to  select 
a system  which  has  been  operational  for  at  least  one  year,  represents  a 
significant  investment  of  time  and  effort,  and  is  of  fundamental  importance 
to  the  organization.  For  this  system  they  answered  38  questions  on  the 
following  topics: 

- name  of  system,  function,  and  end  users 

- number  of  personnel  in  user  groups* 

- number  of  personnel  in  user  groups  actively  involved  in  the 


system  processing  cycle* 


j 


- date  system  became  operational 

- number  of  programs  maintained  and  number  of  source  language 
statements  broken  down  by  language* 

- distribution  of  source  statements  according  to  origination 
year* 

- percentage  of  system  dealing  with  on-line  processing* 

- total  number  of  machine  language  statements* 

- hardware/software  environment  of  system 

- use  of  distributed  processing  and/or  data  base  management 
systems 

- number  of  files,  average  size  of  data  base*,  percentage  of 
data  base  updated  by  time  period* 

- number  and  form  of  predefined  user  reports* 

- productivity  tools  used  in  development 

- time  spent  on  maintenance* 

- division  of  effort  among  types  of  maintenance  activities* 

- percentage  of  maintenance  effort  on  on-line  programs  and  in 
communication  with  user* 

- number  of  people  involved  in  maintenance  of  the  system,  the 
levels  of  their  programming  experience,  when  they  began  to  work 
on  the  system,  and  task  allocation  in  terms  of  analysis  and 
programming 

- formal  procedures  for  maintenance  request  handling,  number 
of  requests  received 

- formal  procedures  for  making  changes  to  programs,  and  number 


of  changes  made 
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- formal  procedures  for  trouble  reporting 

- existence  of  auditing,  documentation,  cost  accounting 
procedures  and  chargeback  methods 

- problem  areas  in  maintenance  of  the  system 

In  the  above  list,  for  the  items  marked  with  an  asterisk  (*)  the  respondents 
also  answered  the  question:  "Check  the  applicable  statement:  the  above 

answer  is:  reasonably  accurate,  based  on  good  data;  a rough 

estimate,  based  on  minimal  data,  or  an  estimate,  not  based  on  any  data." 

The  report  [8]  presents  the  initial  results  of  statistical  analysis.  The 
plan  for  the  future  includes  a revision  of  the  questionnaire  and  survey  to 
a wider  audience.  Respondents  have  been  enthusiastic  about  the  study  and 
have  supported  the  project  financially  by  providing  keypunch  and  computer 
facilities. 

Of  the  respondents  listed  in  [9]  the  distribution  of  allocation  of  effort 
for  maintenance  and  new  development  was 


Maintenance  and  enhancement 

48.0% 

New  development 

46.1% 

Other  activities 

5.9% 

The  breakdown  of  activities  within  maintenance  was 


Category 

Activities 

Relative  Fi 

Corrective 

Emergency  fixes,  routine  debugging 

17.4% 

Adaptive 

Accommodation  of  changes  to  data  inputs 
and  files,  and  to  hardware  and  system 
software 

18.2% 

Perfective 

User  enhancement,  improved  documentation, 
recording  for  computational  efficiency 

60.3% 

Other 

4.1% 

1 * 


Of  the  effort,  perfective  maintenance  is  by  far  the  biggest  area  of  effort. 
This  is  supported  in  Table  I which  indicates  that  user  demands  for  enhance- 
ments and  extensions  are  perceived  by  management  to  be  the  biggest  problem 
area. 


Respondents  were  further  asked  to  rank  possible  problem  areas  in  maintenance. 
This  is  summarized  in  Table  I.  The  table  columns  are  arranged  by  problem 
area,  statistics,  and  relative  frequency.  The  statistics  are  based  on  the 
coding:  1-not  a problem,  2-somewhat  minor  problem,  3-minor  problem, 

4-somewhat  major  problem,  5-major  problem.  Items  marked  with  an  asterisk 
indicate  technical  problem  areas. 


The  predominant  problem  cited  as  more  than  minor  is  that  of  user  demands  for 
enhancements  and  extensions.  Following  this  are  two  technical  issues  (quality 
of  original  system  and  its  documentation)  and  one  management  issue  (competing 
demands  for  personnel  time).  Frequently  mentioned  problems  such  as  hardware 
change,  turnover  of  maintenance  personnel , and  motivation  of  maintenance 
personnel  showed  up  surprisingly  low  (means  of  2.14,  2.46,  and  2.03, 
respectively). 


In  addition  to  the  twenty-four  areas  that  are  mentioned  in  the  questionnaire, 

respondents  were  encouraged  to  list  other  problem  areas.  Areas  mentioned 

included  quality  of  operations  personnel,  turnover  in  user  organization, 

high  learning  curve  due  to  large  system,  and  retaining  personnel  at 

• » 
implementation  time.  ». 

I 
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-TABLE  I PROBLEM  AREAS 
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Problem  Area 

Statist] cs 

Re 

] ative 

Frequency 

Hank 

. . .. 

Mean 

Median 

Std. 

Dev. 

Hot 

Prob. 

Some- 

what 

Minor 

Minor 

Some- 

what 

Major 

Major 

No 

Rosp, 

X- 

User  demands  for  en- 
hancements, extens. 

3.42 

3.72 

1.25 

7.2- 

20.3 

11.6 

36.2. 

18.8 

5.8 

2. 

Quality  of  syst. 
docum. * 

2.99 

3.03 

1 

1.33 

17.4 

15.9 

26.1 

. 20. 3 

14.5  - 

5.8 

3- 

Competing  demands  on 
saint,  personnel 
Jbime  •• 

2.95 

3.00 

1.39 

17.4 

'24.6 

8.7 

29.0 

13.0 

7.2 

4. 

Quality  of  original 
programs* 

2.94 

2.92 

1.42 

20.3 

18.8 

18.8 

18.8 

t 

17.4 

5.8 

• 5- 

Meeting  scheduled 
commitments 

2.79 

2.73 

1.21 

14.5 

26.1 

21.7 

21.7 

7.2 

8.7 

4. 

lack  of  user  under- 
stand. of  syst. 

2.66 

2.53 

1.19 

17.4 

29.0 

• 21.7 

20.3 

5.8 

5.8 

7. 

Availability  of  main, 
program,  personnel 

2.66 

2.53 

1.27 

20.3 

26.1 

21.7 

17.4 

8.7 

5.0 

8. 

Adequacy  of  syst. 
design  spec.* 

2.52 

2.3 

1.37 

29.0 

21.7 

17.4 

14.5 

10.1 

7.2 

9. 

Turnover  of  mainten. 
personnel 

2.46 

2.13 

1.46 

36.2 

17.4 

13.0 

15.9 

11.6 

5.8 

10. 

Unrealistic  user 
expectations 

2.45 

2.50 

1.18 

26.1 

20.3 

29.0 

13.0 

4.3 

7.2 

11. 

Processing  time  of 
system* 

2.31 

2.00 

1.33 

36.2 

20.3 

13.0 

■ 17.4 

5.3 

7.2 

12. 

Forecast,  personnel 
requirements 

2.30 

2.03 

1.28 

33.3 

23.2 

13.0 

17.4 

4.3 

8.7 

13. 

Skills  of  maint. 
-personnel* 

2.20 

1.94 

1.24 

34.8 

26.1 

15.9 

10.1 

5.8 

7.2 

14. 

Changes  to  hardware 
and  software* 

2.14 

1.97 

1.10 

34.8 

26.1 

20.3 

11.6 

1.4 

5.8 

15. 

Budgetary  pressures 

2.09 

1.82 

.1.18 

37.7 

27.5 

11.6  .. 

13.0 

2.9 

7.2 

16. 

Adherence  to  program, 
stds.  in  maint.* 

2.08 

1.94 

1.04 

34.8 

26.1 

23.2 

7.2 

1.4 

7.2 

17. 

Data  integrity* 

2.06 

1.88 

1.12 

34.8 

29.0 

20.3 

1.4 

5.8 

8.7 

18. 

Motivation  of  maint. 
personnel 

2.03 

1.82 

1.10 

37.7 

27.5 

17.4 

7.2 

2.9 

7.2 

19. 

Applic.  run  fail- 
ures* 

2.00 

1.90 

.92 

29.0 

44.9 

13.0 

5.8 

1.4 

5.8 

20. 

Maint.  programming 
productivity 

2.00 

1.07 

.97 

33.  3 

33.3 

15.9 

8.7 

0 

8.7 

21. 

Hard-ware  and  soft- 
ware reliability* 

1.91 

1.76 

.94 

37.7 

33.3 

14.5 

7.2 

0 

7.2 

22. 

Storage  requiremts.* 

1.88 

1.34 

1.24 

55.1 

11.6 

13.0 

8.7 

4. 3 

7.2 

23. 

Mgmt.  Support  of 

system  "•  ‘ . 

1.07 

1.41 

1.17 

49.  3 

17.4 

.11. 6.. 

8.7 

2.9 

10.1 

24. 

Lack  of  user  interest 
in  system 

1.86 

1.50 

1.06 

44.9 

29.0 

11.6 

5.8 

2.9 

5.8 

• • • - r 
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It  is  of  interest  to  determine  if  management  issues  are  more  important  than 
technical  issues.  This  would  serve  as  a guide  in  efforts  to  improve  the 
maintenance  procedures  and  tools.  Statistical  tests  indicate  that  manage- 
ment problems  are  more  significant.  To  carry  out  the  tests 
the  average  rating  was  computed  for  technical  and  management 
areas  for  each  respondent.  The  Mann  Whitney-Wilcoxan  and  sign  tests  were 
selected  to  test  the  hypothesis  that  the  distribution  of  the  average  response 
of  '5ch  category  was  the  same.  These  tests  do  not  depend  on  actual  scores 
but  relative  ratings.  For  the  Mann  Whitney-Wilcoxan  test  the  hypothesis  was 
rejected  at  the  a = .10  level.  For  the  sign  tests  it  was  rejected  at  the 
a = .01  level.  Both  results  indicated  higher  values  for  the  management  areas. 

A second  hypothesis  is  that  the  response  to  the  problem  of  user  demands  for 
enhancement  and  extension  is  significantly  larger  than  average  for  all  problem 
areas.  The  same  nonparametric  tests  were  applied  and  the  hypothesis  of  the 
same  distribution  was  rejected  at  the  a = .10  level.  This  indicates  user 
demands  are  more  of  a problem  than  other  areas. 

The  respondents  were  asked  to  contrast  the  relative  importance  of  main- 
tenance with  new  system  development  within  their  organizations.  The  relative 
frequency  appears  in  Table  II.  It  indicates  most  view  maintenance  as  more 
important  than  new  development.  More  strikingly,  few  view  new  system 
development  as  more  important. 
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TABLE  II 

IMPORTANCE  OF  MAINTENANCE  COMPARED 
TO  NEW  SYSTEM  DEVELOPMENT 


Maintenance  and  Enhancement  Percentage 

By  far  more  important  33.3% 

Somewhat  more  important  21.7 

Equal  importance  34.8 

Somewhat  less  important  5.8 

By  far  less  important  4.3 


Other  findings  are  presented  in  [8].  The  conclusions  based  on  the  limited 
sample  are: 

o Maintenance  and  enhancement  consume  much  of  the  total  resources 
of  systems  and  programming  groups. 

o Maintenance  and  enhancement  tend  to  be  viewed  by  management  as 
at  least  somewhat  more  important  than  new  application  software 
development. 

o In  maintenance  and  enhancement,  problems  of  a management 
orientation  tend  to  be  more  significant  than  those  of  a 
technical  orientation. 

o User  demands  for  enhancements  and  extensions  constitute  the 
most  important  management  problem  area. 

o The  use  of  productivity  aids  in  application  software  development 
remains  limited.  However,  maintenance  programmer  productivity 
is  not  considered  by  management  to  be  a major  problem. 


f ^ 
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Overall  more  attention  should  be  given  to  management  problems  associated  with 
maintenance.  In  practice,  maintenance  work  should  be  categorized  to  permit 
the  gathering  of  more  detailed  management  information.  Project  reporting 
systems  should  be  detailed  with  respect  to  the  type  and  tasks  of  maintenance 
and  enhancement.  This  is  being  done  with  several  organizations  as  a follow- 
up activity. 

A second  major  area  of  effort  has  been  directed  toward  management  trade-offs 
in  the  design  and  evaluation  of  computer  networks.  In  this  area  the  activi- 
ties include  1)  working  with  Mr.  Frank  Miller  of  the  Naval  Electronics 
Laboratory  Center  in  San  Diego  to  perform  trade-off  analyses,  2)  developing 
criteria  for  evaluating  network  services,  and  3)  considering  security  and 
auditing  of  computer  networks.  In  the  first  case,  the  network  model  [10] 
is  anticipated  to  be  installed  soon  at  NELC.  Trade-off  analysis  will  be 
continued  for  several  Naval  networks.  Several  papers  were  prepared  in  the 
area  of  network  evaluation  ([2],  [6]).  These  methods  are  based  on  analyses 
of  network  cost  and  performance  data  using  stochastic  processes.  In  the 
third  case,  an  assessment  of  the  auditing  and  security  problems  was 
developed  in  [6]  and  [7].  The  detailed  results  are  presented  in  the  cited 
paper  references  and  will  not  be  described  here. 

The  work  in  the  next  period  will  continue  the  maintenance  and  network 

research.  In  the  maintenance  area  the  attention  will  be  focused  on 

1)  examining  problem  areas  in  maintenance,  2)  developing  techniques  for 

« 

organizations  to  measure  maintenance  work,  and  3)  developing  methods  for 
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The  people  supported  under  the  contract  were  Professor  Bennet  P.  Lientz, 
Principal  investigator.  Professor  E.  B ”ton  Swanson,  and  the  following 
graduate  students:  Mary  Culnan,  Kweku  Ewusi-Mensah,  James  Schenck,  Ira 
Weiss,  Steven  Kwan  and  Gerry  Tompkins. 

Dr.  Weiss  graduated  with  honors  in  1976  and  joined  the  faculty  of  New  York 
University.  Mr.  Tompkins  is  expected  to  graduate  in  June,  1977  and  enter 
university  teaching.  Mssrs.  Ewusi-Mensah,  Schenck,  and  Kwan  will  be 
advanced  to  Ph.D.  candidacy  by  May,  1977.  Mr.  Kwan's  area  of  interest  is 
computer  network  performance  measurement;  Mr.  Schenck1 s is  minicomputer 
networks;  Mr.  Ewusi-Mensah 's  is  network  service  pricing.  Ms.  Mary  Culnan 
is  a Ph.D.  student  whose  interests  lie  in  data  management  and  computer 
networks. 

Professor  Swanson  served  on  the  School  of  Management  Staffing  Committee  as 
well  as  several  curriculum  committees.  He  published  two  papers  including 
"Computer  Application  System  Development:  Some  Implications  for  Programming 

Practice"  (Data  Management,  May,  1976)  and  "Information  System  Approaches: 
Directions  for  Research  and  Practice" (Management  Datamatics,  1976).  He  also 
presented  the  paper  "The  Dimensions  of  Maintenance"  at  the  Second  Inter- 
national Conference  on  Software  Engineering  in  October  1976.  He  served  as 
a reviewer  for  the  National  Science  Foundation. 

Professor  Lientz  was  advanced  to  Step  II  Associate  Professor.  He  continued 

i 

as  Chairman  of  Computers  and  Information  Systems  and  Coordinator  for 
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Information  Systems  for  the  School.  During  the  year  he  gave  invited  talks 
at  the  Computer  Performance  Symposium,  EDP  Auditors  Annual  Meeting,  and 
the  Annual  Meeting  of  the  ACM.  He  presented  seminars  on  computer  networks 
and  security  to  the  partners  of  Peat,  Marwick,  and  Mitchell.  He  conducted 
a seminar  on  distributed  processing  in  Rio  de  Janeiro,  Brazil.  He  has 
been  listed  in  Who's  Who  in  the  West.  A book  on  the  systems  approach  is 
being  completed  for  Prentice-Hall,  Inc.  A previous  book  entitled  Computer 
Applications  in  Operations  Analysis  (Prentice-Hall,  Inc.)  is  in  its  third 
printing. 

He  continued  as  Associate  Editor,  Computer  Networks  and  as  a reviewer  for 
the  IEEE  Transactions , Management  Science,  and  Journal  of  American  Statistical 
Association. 
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